Assessing Model Fit with Synthetic vs. Real Data

Assessing whether a model is a good fit to the data is non trivial. The standard practice is to compare a few machine learning techniques to learn a model from data, and pick the one with the highest predictive performance. The winner is considered the best fitting model. But each model may involve different machine learning algorithms that carry their own set of parameters and constraints imposed on the corresponding model. This results in a large space in which to explore model performance. The actual best fitting model may have been overlooked due to an unfortunate choice of the algorithm's parameters. We address this issue by complementing performance model comparison with a method that combines real data with synthetic data generated with the competing models. We naturally expect a model to perform best over its corresponding synthetic data, but the analysis across the other synthetic data sets provides some indication of the models generality and robustness under different assumptions about the data. Results of our investigation in the domain of educational data mining show that a model performance is, as expected, best when tested over synthetic data generated aligned with this model. But we observe much greater performance contrasts across synthetic data than across real data. The performance pattern of each model over a given synthetic data set results in a kind of “signature”. We discuss the significance of this signature to assess model fit, and whether it can provide cues to the data's underlying ground truth.